Fusing modalities by multiplexed graph neural networks for outcome prediction from medical data and beyond.

With the emergence of multimodal electronic health records, the evidence for diseases, events, or findings may be present across multiple modalities ranging from clinical to imaging and genomic data. Developing effective patient-tailored therapeutic guidance and outcome prediction will require fusing evidence across these modalities. Developing general-purpose frameworks capable of modeling fine-grained and multi-faceted complex interactions, both within and across modalities is an important open problem in multimodal fusion. Generalized multimodal fusion is extremely challenging as evidence for outcomes may not be uniform across all modalities, not all modality features may be relevant, or not all modalities may be present for all patients, due to which simple methods of early, late, or intermediate fusion may be inadequate. In this paper, we present a novel approach that uses the machinery of multiplexed graphs for fusion. This allows for modalities to be represented through their targeted encodings. We model their relationship between explicitly via multiplexed graphs derived from salient features in a combined latent space. We then derive a new graph neural network for multiplex graphs for task-informed reasoning. We compare our framework against several state-of-the-art approaches for multi-graph reasoning and multimodal fusion. As a sanity check on the neural network design, we evaluate the multiplexed GNN on two popular benchmark datasets, namely the AIFB and the MUTAG dataset against several state-of-the-art multi-relational GNNs for reasoning. Second, we evaluate our multiplexed framework against several state-of-the-art multimodal fusion frameworks on two large clinical datasets for two separate applications. The first is the NIH-TB portals dataset for treatment outcome prediction in Tuberculosis, and the second is the ABIDE dataset for Autism Spectrum Disorder classification. Through rigorous experimental evaluation, we demonstrate that the multiplexed GNN provides robust performance improvements in all of these diverse applications.

Comparison of Chest Radiograph Interpretations by Artificial Intelligence Algorithm vs Radiology Residents.

Importance: Chest radiography is the most common diagnostic imaging examination performed in emergency departments (EDs). Augmenting clinicians with automated preliminary read assistants could help expedite their workflows, improve accuracy, and reduce the cost of care. Objective: To assess the performance of artificial intelligence (AI) algorithms in realistic radiology workflows by performing an objective comparative evaluation of the preliminary reads of anteroposterior (AP) frontal chest radiographs performed by an AI algorithm and radiology residents. Design, Setting, and Participants: This diagnostic study included a set of 72 findings assembled by clinical experts to constitute a full-fledged preliminary read of AP frontal chest radiographs. A novel deep learning architecture was designed for an AI algorithm to estimate the findings per image. The AI algorithm was trained using a multihospital training data set of 342 126 frontal chest radiographs captured in ED and urgent care settings. The training data were labeled from their associated reports. Image-based F1 score was chosen to optimize the operating point on the receiver operating characteristics (ROC) curve so as to minimize the number of missed findings and overcalls per image read. The performance of the model was compared with that of 5 radiology residents recruited from multiple institutions in the US in an objective study in which a separate data set of 1998 AP frontal chest radiographs was drawn from a hospital source representative of realistic preliminary reads in inpatient and ED settings. A triple consensus with adjudication process was used to derive the ground truth labels for the study data set. The performance of AI algorithm and radiology residents was assessed by comparing their reads with ground truth findings. All studies were conducted through a web-based clinical study application system. The triple consensus data set was collected between February and October 2018. The comparison study was preformed between January and October 2019. Data were analyzed from October to February 2020. After the first round of reviews, further analysis of the data was performed from March to July 2020. Main Outcomes and Measures: The learning performance of the AI algorithm was judged using the conventional ROC curve and the area under the curve (AUC) during training and field testing on the study data set. For the AI algorithm and radiology residents, the individual finding label performance was measured using the conventional measures of label-based sensitivity, specificity, and positive predictive value (PPV). In addition, the agreement with the ground truth on the assignment of findings to images was measured using the pooled κ statistic. The preliminary read performance was recorded for AI algorithm and radiology residents using new measures of mean image-based sensitivity, specificity, and PPV designed for recording the fraction of misses and overcalls on a per image basis. The 1-sided analysis of variance test was used to compare the means of each group (AI algorithm vs radiology residents) using the F distribution, and the null hypothesis was that the groups would have similar means. Results: The trained AI algorithm achieved a mean AUC across labels of 0.807 (weighted mean AUC, 0.841) after training. On the study data set, which had a different prevalence distribution, the mean AUC achieved was 0.772 (weighted mean AUC, 0.865). The interrater agreement with ground truth finding labels for AI algorithm predictions had pooled κ value of 0.544, and the pooled κ for radiology residents was 0.585. For the preliminary read performance, the analysis of variance test was used to compare the distributions of AI algorithm and radiology residents' mean image-based sensitivity, PPV, and specificity. The mean image-based sensitivity for AI algorithm was 0.716 (95% CI, 0.704-0.729) and for radiology residents was 0.720 (95% CI, 0.709-0.732) (P = .66), while the PPV was 0.730 (95% CI, 0.718-0.742) for the AI algorithm and 0.682 (95% CI, 0.670-0.694) for the radiology residents (P < .001), and specificity was 0.980 (95% CI, 0.980-0.981) for the AI algorithm and 0.973 (95% CI, 0.971-0.974) for the radiology residents (P < .001). Conclusions and Relevance: These findings suggest that it is possible to build AI algorithms that reach and exceed the mean level of performance of third-year radiology residents for full-fledged preliminary read of AP frontal chest radiographs. This diagnostic study also found that while the more complex findings would still benefit from expert overreads, the performance of AI algorithms was associated with the amount of data available for training rather than the level of difficulty of interpretation of the finding. Integrating such AI systems in radiology workflows for preliminary interpretations has the potential to expedite existing radiology workflows and address resource scarcity while improving overall accuracy and reducing the cost of care.

Extracting and Learning Fine-Grained Labels from Chest Radiographs.

Chest radiographs are the most common diagnostic exam in emergency rooms and intensive care units today. Recently, a number of researchers have begun working on large chest X-ray datasets to develop deep learning models for recognition of a handful of coarse finding classes such as opacities, masses and nodules. In this paper, we focus on extracting and learning fine-grained labels for chest X-ray images. Specifically we develop a new method of extracting fine-grained labels from radiology reports by combining vocabulary-driven concept extraction with phrasal grouping in dependency parse trees for association of modifiers with findings. A total of457finegrained labels depicting the largest spectrum of findings to date were selected and sufficiently large datasets acquired to train a new deep learning model designed for fine-grained classification. We show results that indicate a highly accurate label extraction process and a reliable learning of fine-grained labels. The resulting network, to our knowledge, is the first to recognize fine-grained descriptions offindings in images covering over nine modifiers including laterality, location, severity, size and appearance.

Brain cortical structural differences between non-central nervous system cancer patients treated with and without chemotherapy compared to non-cancer controls: a cross-sectional pilot MRI study using clinically-indicated scans.

BACKGROUND: Cognitive deficit associated with cancer and its treatment is called cancer-related cognitive impairment (CRCI). Increases in cancer survival have made understanding the basis of CRCI more important. CRCI neuroimaging studies have traditionally used dedicated research brain MRIs in breast cancer survivors after chemotherapy with small sample sizes; little is known about other non-central nervous system (CNS) cancers after chemotherapy as well as those not exposed to chemotherapy. However, there may be a wealth of unused data from clinically-indicated MRIs that could be used to study CRCI. OBJECTIVE: Evaluate brain cortical structural differences in those with various non-CNS cancers using clinically-indicated MRIs. DESIGN: Cross-sectional. PATIENTS: Adult non-CNS cancer and non-cancer control (C) patients who underwent clinically-indicated MRIs. METHODS: Brain cortical surface area and thickness were measured using 3D T1-weighted images. An age-adjusted linear regression model was used and the Benjamini and Hochberg false discovery rate (FDR) corrected for multiple comparisons. Group comparisons were: cancer cases with chemotherapy (Ch+), cancer cases without chemotherapy (Ch-) and subgroup of lung cancer cases with and without chemotherapy vs C. RESULTS: Sixty-four subjects were analyzed: 22 Ch+, 23 Ch- and 19 C patients. Subgroup analysis of 16 lung cancer (LCa) patients was also performed. Statistically significant decreases in either cortical surface area or thickness were found in multiple regions of interest (ROIs) primarily within the frontal and temporal lobes for all comparisons. Effect sizes were variable with the greatest seen in the left middle temporal surface area ROI (Cohen's d -0.690) in the Ch- vs C group comparison. LIMITATIONS: Several limitations were apparent including a small sample size that precluded adjustment for other covariates. CONCLUSIONS: Our preliminary results suggest that, in addition to breast cancer, other types of non-CNS cancers treated with chemotherapy may result in brain structural abnormalities. Similar findings also appear to occur in those not exposed to chemotherapy. These results also suggest that there is potentially a wealth of untapped clinical MRIs that could be used for future CRCI studies.

Predicting Meningioma Consistency on Preoperative Neuroimaging Studies.

This article provides an overview of the neuroimaging literature focused on preoperative prediction of meningioma consistency. A validated, noninvasive neuroimaging method to predict tumor consistency can provide valuable information regarding neurosurgical planning and patient counseling. Most of the neuroimaging literature indicates conventional MRI using T2-weighted imaging may be helpful to predict meningioma consistency; however, further rigorous validation is necessary. Much less is known about advanced MRI techniques, such as diffusion MRI, MR elastography (MRE), and MR spectroscopy. Of these methods, MRE and diffusion tensor imaging appear particularly promising.

Nonsuicidal self-injury-related foreign bodies in the emergency department.

Nonsuicidal self-injury is a common cause of foreign bodies encountered in emergency departments, resulting in significant morbidity, mortality, and economic costs. We review the role of imaging and frequent imaging findings in nonsuicidal self-injury, illustrated by case examples from a level 1 trauma center with dedicated jail and psychiatric wards in a major US metropolitan teaching hospital.

Concurrent intracranial and spinal subdural hematoma in a teenage athlete: a case report of this rare entity.

A 15-year-old male high school football player presented with episodes of headache and complete body stiffness, especially in the arms, lower back, and thighs, immediately following a football game. This was accompanied by severe nausea and vomiting for several days. Viral meningitis was suspected by the primary clinician, and treatment with corticosteroids was initiated. Over the next several weeks, there was gradual symptom improvement and the patient returned to his baseline clinical status. The patient experienced a severe recurrence of the previous myriad of symptoms following a subsequent football game, without an obvious isolated traumatic episode. In addition, he experienced a new left sided headache, fatigue, and difficulty ambulating. He was admitted and an extensive workup was performed. CT and MRI of the head revealed concurrent intracranial and spinal subdural hematomas (SDH). Clinical workup did not reveal any evidence of coagulopathy or predisposing vascular lesions. Spinal SDH is an uncommon condition whose concurrence with intracranial SDH is an even greater clinical rarity. We suggest that our case represents an acute on chronic intracranial SDH with rebleeding, membrane rupture, and symptomatic redistribution of hematoma to the spinal subdural space.

MRI appearances of ovarian serous borderline tumor: pathological correlation.

PURPOSE: To investigate the spectrum of MRI appearances of ovarian serous borderline tumor (SBT). MATERIALS AND METHODS: Following ethics approval, 31 patients with 51 histologically proven ovarian SBTs underwent preoperative MRI. Images were evaluated, by two observers for the location, shape, size, internal architecture, signal intensity, and extent or stage of the tumors. The MRI findings were correlated with pathological findings. RESULTS: Twenty of 31 patients (65%) demonstrated bilateral ovarian SBTs on MRI. Three MRI morphological patterns of ovarian SBT were identified: (i) Mainly cystic mass with multiple intracystic papillary projections from the wall and septations was observed in 24 (47%) tumors. (ii) Solid mass with hierarchical branching papillary and fibrous stalk architecture was observed in 8 (16%) tumors. The branching papillary projections were hyperintensity on T2WI, intermediate intense on DWI, and enhanced intensely after the administration of Gd-DTPA. The internal branching fibrous stalks were hypointensity on T2WI and enhanced slightly. (iii) Mixed cystic-solid mass was observed in 19 (37%) tumors. The cystic and solid components had the architecture and signal intensity similar to those of cystic and solid SBTs. Papillary projections were the common architecture of all three types of tumors. CONCLUSION: On MRI, the ovarian SBT has some morphological distinguishing features. The solid papillary architecture with internal branching fibrous stalk is a somewhat more characteristic MRI appearance.

Image coregistration: quantitative processing framework for the assessment of brain lesions.

The quantitative, multiparametric assessment of brain lesions requires coregistering different parameters derived from MRI sequences. This will be followed by analysis of the voxel values of the ROI within the sequences and calculated parametric maps, and deriving multiparametric models to classify imaging data. There is a need for an intuitive, automated quantitative processing framework that is generalized and adaptable to different clinical and research questions. As such flexible frameworks have not been previously described, we proceeded to construct a quantitative post-processing framework with commonly available software components. Matlab was chosen as the programming/integration environment, and SPM was chosen as the coregistration component. Matlab routines were created to extract and concatenate the coregistration transforms, take the coregistered MRI sequences as inputs to the process, allow specification of the ROI, and store the voxel values to the database for statistical analysis. The functionality of the framework was validated using brain tumor MRI cases. The implementation of this quantitative post-processing framework enables intuitive creation of multiple parameters for each voxel, facilitating near real-time in-depth voxel-wise analysis. Our initial empirical evaluation of the framework is an increased usage of analysis requiring post-processing and increased number of simultaneous research activities by clinicians and researchers with non-technical backgrounds. We show that common software components can be utilized to implement an intuitive real-time quantitative post-processing framework, resulting in improved scalability and increased adoption of post-processing needed to answer important diagnostic questions.

Magnetic resonance imaging confirmation of resolution of periodontoid pannus formation following C1/C2 posterior transarticular screw fixation.

Chronic odontoid fractures are considered unstable spinal lesions. Chronic instability in this region leads to the development of an inflammatory pannus, which can progress resulting in spinal cord compression radiographically and a myelopathy syndrome clinically. In this report we document three cases of reversal of pannus after C1/C2 transarticular screw fixation of an unstable odontoid fracture. Three patients were identified with chronic odontoid fractures and spinal cord compression due to periodontoid pannus formation. All patients presented with a progressive myelopathy syndrome. All patients underwent preoperative and postoperative magnetic resonance imaging (MRI) of the craniovertebral junction. C1/C2 transarticular screw fixation was performed for stabilization of C1/C2. Postoperatively there were no complications. Postoperative MRI at 6 months demonstrated resolution of the ventral pannus. Moreover, all patients exhibited improvement of preoperative neurological deficits. MRI is the imaging technique of choice for diagnosis and follow-up of patients with chronic odontoid fractures and ventral pannus. C1/C2 transarticular screw fixation provides a viable method for spinal stabilization in this region. In addition, stabilization can result in resolution of inflammatory pannus formation secondary to instability of the C1/C2 articulation.